Repository navigation
fix(refcount): never read or write past the refcount region - #39
Merged
Merged
Conversation
The refcount region is 1 MB, which holds entries for 130816 blocks (127.75 GiB). Region bounded block numbers only by the volume's block count, so on larger volumes the sectors of higher blocks fell in the snapshot registry and, past about 255.5 GiB, in data blocks. Registry sectors use the same CRC32C format, so a snapshot or clone of a high block, or a plain delete, rewrote registry bytes without an error. NewRegion now takes the region size. Blocks past it read as having a single owner and are never written; sharing them fails with ErrBlockPastRegion. Scan and Initialize stay inside the region, and mount warns when the volume has more blocks than the region covers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shareInto marked the source shared before Share refused blocks past the refcount region, so after a refusal every write to the file took the copy-on-write check. Check that the region has entries for all source blocks first, and report the refusal as ENOTSUP like uncounted sharing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 10, 2026
…he device (#47) * feat(superblock)!: size the bitmap, refcount and inode regions from the device Layout v2 gave the block bitmap, refcount table and inode table 1 MB each. That holds 4096 inodes, and refcounts for blocks past 128 GB landed in the snapshot registry and then in data blocks; the bitmap overflows into the inode table past 8 TB. Layout v3 puts the fixed-size regions first and sizes the other three at format time: the bitmap and refcount table cover every block the device can hold, and the inode table gets one inode per -bytes-per-inode bytes (default 1 MB, one per data block), between 4096 and 16M inodes. Mount checks the region order and that each region is big enough for the stored block and inode counts, so a grown device still mounts. Mount and fsck refuse v2 volumes and say to copy the data off and reformat. The inode table is now up to 4 GB, so FindFreeSlot and FreeCount (statfs) read it 1 MB at a time instead of one sector per inode. fsck reports a refcount table too small for the volume's blocks. BREAKING CHANGE: v2 volumes no longer mount; reformat with -format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(integration): cover layout v3 with 4200 files and a 200 GB volume TestMoreThan4096Files formats with -bytes-per-inode 65536 and keeps 4200 files across a remount. TestLargeVolumeSnapshotsHighBlocks fallocates past block 130816, where v2's refcount table ended, then snapshots, copies on write and clones data there on a 200 GB sparse image. Both end with a clean fsck. TestMountRefusesLayoutV2 checks that mount and fsck say to reformat a v2 volume. FormatAndMount now passes ExtraArgs too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: create 4200 files from every node, also in Docker and VM A create costs about 130 ms, so TestLargeVolume has four nodes fill their own directories at once, on the 200 GB image, and skips in -short mode; it replaces TestMoreThan4096Files and TestLargeVolumeSnapshotsHighBlocks. Docker and VM clusters now format with -bytes-per-inode 65536 (12288 and 16384 inodes), and test_more_than_4096_files creates and removes 4200 files there the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(refcount): test the region bound without the v2 layout Layout v3 sizes the refcount table for every block, so mount can no longer see a table smaller than the volume; drop the warning #39 added for that. Its test now builds an undersized region directly instead of formatting a 200 GB v2 volume. The superblock package computes sector capacities itself and a test checks them against bitmap and refcount, because a refcount test imports superblock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(superblock): check the grown device's Close error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fsck): report a refcount table smaller than the volume once, as an error #39 already reports a table without entries for every block, as a warning, since v2 volumes had one. Drop the second check added with v3 and make #39's an error: v3 mount refuses such a volume. Its test now shrinks the table in the superblock, since v3 never formats one that small. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(test): wait for other nodes to see the directories in the 4200-file test Docker run on 4455dfb: node 2 could not cd into a directory node 1 had just made, because its attribute cache still held the old listing, so 1100 files were never created. Each node now polls up to 60 s for its directories before creating files, and the final count polls up to 60 s instead of sleeping 1.5 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(test): point the refcount bound scenario at its renamed test CI's parity check failed: the manifest still named TestBlocksPastRegionLeaveTheSnapshotRegistryAlone. Also describe both refcount-past-region scenarios the way their tests now set them up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(docker): run the 4200-file test in its own group It took 13m38s in CI, which pushed group 1 to 19m56s against the orchestrator's 20-minute wait. Alone in group 3 it runs next to the other two groups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(superblock): bound region offsets and recognize a destroyed v3 superblock Review of #47 found two ways the v3 checks fail open: - CheckLayout added region sizes without bounding them, so a hand-made superblock with a bitmap size near 2^64 wrapped the next offset back to 1 MB and put the refcount table on the node registry. Each region must now end inside the device, and InodeCount must fill the inode table exactly. The fuzzer runs CheckLayout and checks that accepted regions are disjoint and inside the device. - A v3 device with an overwritten superblock was reported as unformatted, telling the operator to run -format. The old probe looks for an inode table at 2 MB, where v3 keeps lock records. Node registry slots at 1 MB now identify a v3 device, and mount says not to format it. Format refuses a -bytes-per-inode that leaves no data blocks instead of writing a volume the next mount refuses, and the v1 and v2 version errors now explain each version separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fsck): report layouts that mount refuses fsck accepted superblocks mount refuses, such as a swapped bitmap and refcount table, an inode count below the table, or a gap before the data. Phase 1 now runs the same CheckLayout as mount and reports a failure as an error, then keeps checking. The fsck unit tests build real v3 superblocks instead of a hand-made 16-node layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(inode): count free inodes without the table lock Every statfs, not only df -i, scans the whole inode table: 50 MB at 200 GB. Holding t.mu for that stalled inode writes on this node, and other nodes change the table without it anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: poll for other nodes' files in TestLargeVolume It slept 1.5 s before counting files the other nodes made; the Docker and VM versions already poll. Also shorten the comments this PR added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fsck): refuse to repair a layout mount refuses With the bitmap and refcount offsets swapped, phase 5 reads the real bitmap as refcount sectors, which pass the same checksum, and repair rewrites them as refcounts. Repair now refuses (exit 8) after phase 1 reports a layout error, and a dry run says it would refuse. The destroyed-superblock message now says -format is fine when the device was wiped on purpose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On volumes over about 128 GiB, refcount updates for high blocks wrote past the 1 MB refcount region. A snapshot, a clone or a plain delete of a file with data above block 130816 silently rewrote snapshot registry bytes, because registry sectors use the same CRC32C format. Above about 255.5 GiB the writes land in data blocks.
Proven on main with a 200 GiB sparse image:
Increment(130816)succeeded and wrote over the registry's first sector.Releasechanged a registry descriptor without returning an error.The new test fails nine ways on main.
Now blocks past the region read as single-owner. Snapshots and clones of them fail with
ErrBlockPastRegion, and deletes keep working. The real fix, sizing the region at format time, is a layout change that will be folded into the v3 layout work.Refused snapshots and clones return ENOTSUP and leave the source file unmarked.
Volumes used with older builds: on a volume over about 128 GiB where snapshots or clones were taken before this change, counts for blocks 130816 and up went into the snapshot registry. Those blocks now read as having one owner, so deleting one copy of such a file can free a block the other copy still maps. fsck can't see this yet; the follow-up PR adds that check.
🤖 Generated with Claude Code